Asymptotically optimal priority policies for indexable and non-indexable restless bandits

We study the asymptotic optimal control of multi-class restless bandits. A restless bandit isa controllable stochastic process whose state evolution depends on whether or not the bandit ismade active. Since finding the optimal control is typically intractable, we propose a class of prioritypolicies that are proved to be asymptotically optimal under a global attractor property and a technicalcondition. We consider both a fixed population of bandits as well as a dynamic population wherebandits can depart and arrive. As an example of a dynamic population of bandits, we analyze amulti-class M/M/S+M queue for which we show asymptotic optimality of an index policy.We combine fluid-scaling techniques with linear programming results to prove that when banditsare indexable, Whittle’s index policy is included in our class of priority policies. We thereby generalizea result of Weber and Weiss (1990) about asymptotic optimality of Whittle’s index policy to settingswith (i) several classes of bandits, (ii) arrivals of new bandits, and (iii) multiple actions.Indexability of the bandits is not required for our results to hold. For non-indexable bandits wedescribe how to select priority policies from the class of asymptotically optimal policies and presentnumerical evidence that, outside the asymptotic regime, the performance of our proposed prioritypolicies is nearly optimal.